Surgical suite integration and optimization

ABSTRACT

Systems, methods, and computer-readable media for integrating and optimizing a surgical suite. An ophthalmic suite can include a surgical console, a heads-up display communicatively coupled with a surgical camera for capturing a three-dimensional image of an eye, and a surgical suite optimization engine. The surgical suite optimization engine can performs a wide variety of actions in response to action codes received from the other components in the surgical suite. The surgical suite optimization engine can be integrated within another component of the surgical suite, can be a stand-alone module, and can be a cloud-based tool.

BACKGROUND Field of the Disclosure

The present disclosure relates to ophthalmic surgery, and morespecifically, to systems, methods, and computer-readable media forintegrating and optimizing a surgical suite.

Description of Related Art

Surgical consoles, imaging devices, laser devices, diagnostic devices,and other accessories that are used in ophthalmic surgery can generate,record, and transmit data relating to features of a surgery, proceduresteps and stages, etc. However, the separate devices that make up asurgical suite are currently not adequately connected.

Furthermore, ophthalmic surgery is commonly performed using an operatingmicroscope to visualize various structures in the eye. However, manyophthalmic surgeries take a considerable amount of time. For example,vitreoretinal surgery can last over three hours in length. Over thecourse of these long surgeries, surgeons can become fatigued after beingbent over a microscope eyepiece. A high dynamic range digital camerasystem can be used to capture a multidimensional visualization of apatient's eye that can be transmitted to a high-definition display.While such displays allow a surgeon to conduct surgery in a heads-upposture, the potential for using stereoscopic visualization on aheads-up display capable of performing advanced operations for improvingother areas of a surgical practice is not currently realized.

SUMMARY

The disclosed embodiments of the present technology relate to systems,methods, and computer-readable media are disclosed for integrating andoptimizing a surgical suite. Some embodiments involve an ophthalmicsystem that includes a surgical console and a heads-up displaycommunicatively coupled with a surgical camera for capturing athree-dimensional image of an eye. The heads-up display can display astereoscopic representation of the three-dimensional image of the eyeand a viewer can wear three-dimensional glasses to view thethree-dimensional image of the eye.

The surgical console can have tools for conducting an ophthalmicprocedure and can monitor quantitative features of the ophthalmicprocedure. The surgical console can determine to transmit an action coderelating to the quantitative feature of the ophthalmic procedure in avariety of circumstances.

The ophthalmic system also includes a surgical suite optimization enginethat performs a wide variety of actions in response to action codesreceived from the surgical console or other components of a surgicalsuite. In some cases, the surgical suite optimization engine can beintegrated in the heads-up display, integrated within another componentof the surgical suite, a stand-alone module, a cloud-based tool, etc.

The surgical console can monitor and/or transmit, to the surgical suiteoptimization engine, procedure data describing a plurality of proceduresteps for a plurality of ophthalmic procedures. In some cases, thesurgical console monitors a control limit for an aspect of one or moreprocedure step and can issue an action code when the aspect of the oneor more procedure step comes within a predetermined proximity to thecontrol limit. In these cases, the surgical suite optimization enginecan perform an action of displaying an alert relating to the proximityto the control limit.

In some cases, the procedure data can also describe a plurality ofsurgical phases and/or procedure steps. In some cases, the surgicalconsole issues an action code at a transition between surgical stepsand/or phases. The surgical suite optimization engine can perform avariety of actions when it receives the action code indicating atransition between steps and/or stages including, but not limited to:automatically adjusting display parameters of the stereoscopicrepresentation of the three-dimensional image of the eye; displaying oneor more overlay over the stereoscopic representation of thethree-dimensional image of the eye; filtering color attributes of thestereoscopic representation of the three-dimensional image to highlightone or more areas of the eye in the stereoscopic representation of thethree-dimensional image; enhancing image features of the stereoscopicrepresentation of the three-dimensional image; moving, with a roboticarm, the heads-up display relative to the position of the surgeon at thetransition between surgical phases to optimize a stereoscopic viewingdistance of the first region of the eye to a position to optimize astereoscopic viewing distance of the second region of the eye; causing adiagnostic imaging device to focus on an area of the eye.

The ophthalmic system can also include a laser treatment component andthe surgical suite optimization engine can perform, at a transitionpoint between laser treatment steps, digital signal processing tomaximize red contract of the stereoscopic representation of thethree-dimensional image of the eye when the laser treatment transitionsfrom the emission of the treatment beam to the emission of the aimingbeam and to neutralize a green flashback from a retina in thestereoscopic representation of the three-dimensional image of the eyewhen the laser treatment transitions from the emission of the aimingbeam to the emission of the treatment beam.

In some cases, the procedure data can also describe user preference datadescribing a surgeon's or other surgical staff member's preference forone or more procedure step for an ophthalmic procedure.

In some cases, the surgical console can monitor an operational status ofa component of the surgical console and can issue an action code whenthe component malfunctions. The surgical suite optimization engine canreceive the action code describing the malfunction and automaticallyestablish a video conference connection with a technical support center.

In some cases, the heads-up display can display a dashboard thatincludes a main surgical view window displays the stereoscopicrepresentation of a three-dimensional image of an eye along with avariety of menus, icons, picture-in-picture displays, graphical overlayson top of the image of the eye, etc. The stereoscopic representation ofthe three-dimensional image of the eye can also be digitally adjusted tofocus on certain anatomy, to sharpen certain regions, to change color ofthe image, etc.

In some cases, the dashboard can be controlled by a touchpad interface,a touchscreen interface, a voice control interface, etc. In some cases,the three-dimensional glasses can contain a motion tracker tracksmovement of the wearer's gaze and can transmit a gaze signal to thesurgical suite optimization engine. The surgical suite optimizationengine can use the gaze signal to display and move a pointer on thedisplay and activate functions of the dashboard tools, menus, etc.

The surgical suite optimization engine integrates and synergisticallyoptimizes a wide variety of surgical functions. The surgical suiteoptimization engine can also include a transcode engine for interpretingcommunications from a surgical suite, machine learning and artificialintelligence modules for learning how to optimize procedure to maximizepatient outcome, speech recognition modules to learn and disambiguatevoice inputs into known voice commands.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present technology, itsfeatures, and its advantages, reference is made to the followingdescription, taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 illustrates an ophthalmic suite including a surgical console, aheads-up display, and a surgical camera system;

FIGS. 2A and 2B illustrate methods of improving a surgical procedureusing surgical suite optimization engine;

FIGS. 3A-3C illustrate a surgical suite optimization engine running on aheads-up display in an ophthalmic surgery suite

FIG. 4A illustrates representation of a graphical user interface of adisplay dashboard that can be displayed on a heads-up display in asurgical suite;

FIG. 4B illustrates a method of gaze-enabled control of a graphical userinterface;

FIG. 5A illustrates a method of adjusting display settings based onsurgical progression;

FIGS. 5B-5E illustrate examples of display settings for various steps ofan ophthalmic procedure;

FIGS. 6A and 6B illustrate a system for adjusting the distance of aheads-up display in relation to an observation point;

FIG. 6C illustrates a method of optimizing stereopsis for surgical staffviewing the displayed stereoscopic representation of a three-dimensionalimage of eye anatomy in response to a change in the phase of a surgicalprocedure and/or procedure type;

FIG. 7 illustrates a method of ensuring centration of a surgical cameraon an eye;

FIG. 8 illustrates a method of enhancing diagnostic visualization forsurgical staff viewing a stereoscopic representation of athree-dimensional image of eye anatomy;

FIG. 9 illustrates a method of a surgical suite optimization enginedynamically adjusting anatomical area of focus, anatomical highlighting,and graphical overlays;

FIG. 10 illustrates a method of monitoring control limits and providingsurgical alerts and recommendations;

FIG. 11A illustrates a method of modulating color characteristics of astereoscopic representation of a three-dimensional image of eye anatomyto optimize visualization during a laser treatment step of an ophthalmicprocedure

FIGS. 11B and 11C illustrate examples of graphical user interfacesshowing optimized visualization for two laser states according to someembodiments of the present technology;

FIG. 12 illustrates a method of identifying a problem in the surgicalsuite and automatically initiating contact with a support professionalto remedy the problem;

FIG. 13 illustrates a method of a surgical suite optimization enginemaintaining appropriate inventory for a surgical practice; and

FIG. 14A and FIG. 14B illustrate exemplary possible system embodiments.

DESCRIPTION

Systems, methods, and computer-readable media are disclosed forintegrating and optimizing a surgical suite. In some embodiments of thedisclosed technology, surgical consoles, imaging devices, laser devices,diagnostic devices, and other accessories that are used in ophthalmicsurgery are integrated in an inter-networked surgical suite. Theinter-networked surgical suite can include a surgical camera thatgenerates high-definition images of eye anatomy and transmits the imagesto a heads-up display for three-dimensional stereoscopic visualizationof the anatomy. Some embodiments of the disclosed technology involve asurgical suite optimization engine for performing advanced operationsfor improving surgical practice and patient outcome.

FIG. 1 illustrates an ophthalmic suite 100 including a surgical console102, a heads-up display 104, and a surgical camera system 106 thatpositions a surgical camera 108 over a patient table 110. The surgicalcamera 108 can be a High Dynamic Range (HDR) camera with a resolution,image depth, clarity and color contrast that enables a high quality,three-dimensional image of patient anatomy. For example, the presentdisclosure describes an HDR camera used to resolve high quality,three-dimensional views of an eye as well as surgical consoles andheads-up displays for performing actions during ophthalmic proceduresthat use the HDR camera. However, those with ordinary skill in the arthaving the benefit of the present disclosure will readily appreciatethat the disclosed technology can be applied to various other fields ofanatomical diagnostics, surgery, etc.

The surgical camera 108 can be communicatively coupled with the heads-updisplay 104 (e.g. via a wired connection, a wireless connection, etc.)and the heads-up display 104 can display a stereoscopic representationof the three-dimensional image providing a surgeon, staff, students, andother observers depth perception into the eye anatomy. The surgicalcamera 108 can also be used to increase magnification of the eye anatomywhile maintaining a wide field of view. The stereoscopic representationof the three-dimensional image can be viewed on the heads-up displaywith stereoscopic glasses, as an autostereogram, using Fresnel lenses,etc. With the stereoscopic representation of the three-dimensional imagedisplayed on the heads-up display 104, a surgeon can perform procedureson a patient's eye while in a comfortable position (e.g. sitting on astool 112) without bending over a microscope eyepiece and straining hisneck.

The surgical console 102 can be communicatively coupled with theheads-up display 104 and/or the surgical camera system 106. In someembodiments, the heads-up display 104 can receive information (e.g.surgical parameters) from the surgical console 102 and display theinformation on the heads-up display 104 along with the stereoscopicrepresentation of the three-dimensional image. The surgical console 102can also send signals to the heads-up display 104 for performingoperations (e.g. starting and stopping video recording).

In addition, ophthalmic suite 100 can also include a surgical suiteoptimization engine (not shown) containing one or more processors (notshown) and memory (not shown) for performing advanced operations. Asexplained in greater detail below, the surgical suite optimizationengine can integrate a surgical suite and can perform a wide variety ofactions to synergistically optimize a wide variety of surgicalfunctions. For example, the surgical suite optimization engine can:enable gaze-tracking to navigate a display dashboard; perform digitalsignal processing for adjusting display settings, applying filters,increasing contrast, neutralizing particular wavelengths, identifyinganatomy, etc.; display of a wide variety of images at various zoomdepths, menus, other diagnostic images, surgical productivityapplications, surgical schedules, live video teleconferencing, etc.;control robotic arms to move the heads-up display to optimize stereopsisor ensure centration of the surgical camera; control a diagnostic deviceto automatically focus on anatomy based on the procedural step; providealerts and recommendations to surgical staff; modulate color effectsduring laser treatment; etc.

Also, surgical suite optimization engine can include a network interfacethat allows the surgical suite optimization engine to communicate withthe surgical camera system 106, the surgical console 102, and othersurgical systems. In some cases, the surgical suite optimization enginecan serve to inter-network some or all of network-capable components ina surgical suite 100.

Inter-networking the surgical suite can allow for advancements insurgical practice and patient outcome. For example, the surgical console102 can include a wide assortment of tools for performing variousaspects of ophthalmic procedures and can include memory (not shown) andone or more processors (not shown) for controlling the tools as well asmonitoring a wide variety of quantitative features of a surgicalprocedure. When the surgical console 102 is connected with the surgicalsuite optimization engine, the surgical console 102 send informationabout the quantitative features of a surgical procedure (e.g. via one ormore action codes) to the surgical suite optimization engine. Inresponse, the surgical suite optimization engine can interpret theinformation from the surgical console 102 perform actions that improveother areas of a surgical procedure or a surgical practice. Moregenerally, the surgical suite optimization engine can receive, via thenetwork interface, an assortment of information from any of componentsin the surgical suite 100 and, in response to the gathered information,perform actions that improve a wide variety of areas of a surgicalprocedure or a surgical practice, resulting in better patient outcomes.

FIG. 2A illustrates a method 200 of improving a surgical procedure usinga surgical suite optimization engine. The method 200 involves arrangingan improved surgical suite with a heads-up display containing a surgicalsuite optimization engine, a surgical camera, and a surgical console205. In some cases, the heads-up display, the surgical camera, and thesurgical console are communicatively coupled via a wireless connection.

The method 200 further involves the surgical camera capturing athree-dimensional image of an eye 210 and transmitting thethree-dimensional image of the eye to the heads-up display 215. Next,the method 200 involves displaying a stereoscopic representation of thethree-dimensional image of the eye on the heads-up display 220.

Also, the method 200 involves monitoring a quantitative feature of anophthalmic procedure 225 and the surgical console determining totransmit an action code to the surgical suite optimization engine of theheads-up display 230.

FIG. 2B illustrates another method 235 of improving a surgical procedureusing surgical suite optimization engine. The method 235 can alsoinvolve the surgical suite optimization engine receiving the action code240 and performing an action 245 in response to the action code. Asexplained above, a surgical console can monitor a wide variety ofquantitative features of ophthalmic procedures and can process a varietyof instructions for determining when to transmit an action code.Likewise, the surgical suite optimization engine can perform a widevariety of actions in response to receiving the action codes. Specificexamples of monitoring a procedure, determinations by the surgicalconsole to issue action codes, and specific actions by the surgicalsuite optimization engine are described in greater detail below.However, those with ordinary skill in the art having the benefit of thepresent disclosure will readily appreciate that a wide variety of otheraspects of a surgical practice can benefit from the disclosedtechnology.

FIG. 3A illustrates a surgical suite optimization engine 315 running ona heads-up display 304 in an ophthalmic surgery suite 300. Theophthalmic surgery suite 300 can also include a surgical camera 306 andone or more surgical console(s) 302. The surgical camera 306 and thesurgical console(s) 302 are communicatively coupled with the surgicalsuite optimization engine 315. The surgical suite optimization engine315 includes a processor 323 and memory 321 that contains instructionsfor performing a variety of actions. In some cases, the surgical suiteoptimization engine includes an additional graphical processing unit(not shown) for performing digital image post-processing and otheradvanced graphical operations. The surgical suite optimization engine315 can also contain a communication interface 317 that can wirelesslycommunicate with the surgical camera 306, the surgical console(s) 302,and other devices. In some cases, the surgical suite optimization engine315 is communicatively coupled with a cloud-based Machine Learning andArtificial Intelligence engine 328.

The surgical camera 306 can capture an image of a patient's eye. In somecases, the surgical camera 306 is a HDR that captures athree-dimensional image of the patient's eye. The surgical camera 306can transmit the three-dimensional image of the patient's eye to theheads-up display 304 and the heads-up display 304 can display astereoscopic representation of the three-dimensional image of the eye ina Graphical User Interface 319. The stereoscopic representation of thethree-dimensional image of the eye can be viewed by a surgeon, surgicalstaff, students, etc. using 3D glasses 320. In some cases, the heads updisplay can comprise a high definition wide-screen display, a computerdisplay, a tablet, a smartphone, etc. The heads-up display 304 can alsoinclude touchscreen capability, voice control capability, gaze-enabledcontrol (as described below), etc.

The surgical console(s) 302 can include tools for conducting a varietyof ophthalmic procedures. For example, the surgical console(s) 302 caninclude tools for performing cataract surgery, inter-ocular (IOL) lensplacement, vitreoretinal surgery, glaucoma surgery, laser refractivesurgery, etc. Although specific examples of ophthalmic surgeries arelisted herein, those with ordinary skill in the art having the benefitof the present disclosure will readily recognize that a wide variety ofsurgery types (ophthalmic or otherwise) can benefit from the presenttechnology. The surgical console(s) 302 can also include a processor(not shown) and memory (not shown) containing instructions that, whenexecuted by the processor, cause the surgical console(s) 302 to operatethe tools, monitor features of the surgery, guide a surgical staffthrough steps and stages of a surgical procedure, etc. Further, asexplained in more detail below, the processor, cause the surgicalconsole(s) 302 to determine when to transmit an action code relating toa feature of an ophthalmic procedure to the surgical suite optimizationengine 315.

The surgical console(s) 302 can also be coupled with a laser module 322and a variety of surgical accessories 324 (e.g. wireless footswitches,handpieces, probes, preloaded IOL delivery systems, aspirators,illuminators, diagnostic devices, micro-stent delivery systems, etc.)for performing surgical procedures. In some cases, the surgicalaccessories contain an integrated communication interface (not shown)for communicating with the surgical suite optimization engine 315,either directly or through another component in the surgical suite 300.For example, the surgical suite 300 can include a footswitch (not shown)that can be used to start and stop a video recording feature on theheads-up display 304. Also, as explained in greater detail below, thesurgical console(s) 302 can also be coupled with a variety of otherdevices and applications such as diagnostic devices 314, surgicaltraining applications 316, surgical practice administration applications318, etc.

In some cases, some or all of the surgical console(s), surgical camera306, heads-up display 304, laser module 322, surgical accessories 324,diagnostic device(s) 314, surgical training applications 316, surgicalpractice administration applications 318, and the surgical suiteoptimization engine 315 are connected as an inter-network of “smart”devices, aka connected as an Internet of Things (IoT). In some cases,the surgical suite optimization engine 315 optimizes the performance ofthe ophthalmic surgery suite 300 by ingesting data from inter-networkeddevices and controlling other aspects of the ophthalmic surgery suite300. In some cases, the surgical suite optimization engine 315 includesa transcode engine 325 for performing a variety of transcoding functionssuch as one or more of recognizing signals from connected devices,translating the signals, converting signals to a common intermediatelanguage, etc.

In some cases, the surgical suite optimization engine 315 can include avoice control module 330. The voice control module 330 can receive andprocess voice commands to control an aspect of the surgery (e.g.navigation of a graphical user interface, control of the surgicalconsole, control of settings, etc.) In some cases, the voice controlmodule 330 can recognize a speaker of voice commands and can allow onlycertain speakers (e.g. surgeon, attending nurse, etc.) to controlcertain aspects of surgery. Also, in some cases, the voice controlmodule 330 can disambiguate voice inputs (e.g. common parlance) intorecognized voice commands. Similarly, the voice control module 330 cancommunicate with the machine learning and artificial intelligence engine328 and can learn speech patterns, new vocabulary, etc.

The surgical suite optimization engine 315 can monitor the ophthalmicsurgery suite 300, can receive signals (e.g. an action code from thesurgical console(s) 302) from connected devices, and can perform avariety of actions to optimize the ophthalmic surgery suite 300. Also,the surgical suite optimization engine 315 can include a displayoptimization database 327 and a user profile database 329 to furtheroptimize display settings for that particular surgical step, asexplained in greater detail below.

As shown in FIG. 3A, the surgical suite optimization engine 315 isintegrated in the heads-up display 104. However, the surgical suiteoptimization engine 315 can be part of a surgical console, a laserconsole, a diagnostic device, a surgical camera, etc. Also, the surgicalsuite optimization engine 315 can be a stand-alone device, a cloud-basedprocessing engine, etc.

FIG. 3B illustrates a stand-alone surgical suite optimization engine315′ in a surgical suite 300′. The surgical suite optimization engine315′ can be communicatively coupled with the surgical console(s) 302,laser module 322, surgical accessories 324, surgical camera 306,heads-up display 304, cloud-based machine learning and artificialintelligence engine 328, diagnostic device(s) 314, surgical trainingapplications 314, surgical practice applications 318, etc.

FIG. 3C illustrates a cloud-based surgical suite optimization engine315″ connected with a surgical suite 300″. The surgical suiteoptimization engine 315″ can be communicatively coupled with thesurgical console(s) 302, laser module 322, surgical accessories 324,surgical camera 306, heads-up display 304, cloud-based machine learningand artificial intelligence engine 328, diagnostic device(s) 314,surgical training applications 314, surgical practice applications 318,etc.

Like the surgical suite optimization engine 315 of FIG. 3A, thestand-alone surgical suite optimization engine 315′ and the cloud-basedsurgical suite optimization engine 315″ can monitor the ophthalmicsurgery suite 300′, 300″ and perform actions to optimize the surgicalsuite and patient outcomes.

FIG. 4A illustrates representation of a graphical user interface (GUI)400 of a display dashboard that can be displayed on a heads-up displayin an ophthalmic suite according to some embodiments of the presenttechnology. The GUI 400 can include a main surgical view window 402 thatdisplays a stereoscopic representation of a three-dimensional image ofan eye received from the surgical camera. Also, the image of the eye canshow the tools 404, 406 that are being used to perform surgery. In somecases, the stereoscopic representation of the three-dimensional image ofthe eye can be digitally adjusted to focus on certain anatomy, tosharpen certain regions, to change color of the image, etc. Also, avariety of graphical overlays 408, 410, 412, 414, 416 can be displayedalong with the image of the eye. In some cases, the surgical suiteoptimization module can identify certain anatomy (e.g. the macula) andan anatomical identification overlay 408 can be displayed over the imageof the eye. In some cases, a surgical planning application and/or thesurgical suite optimization engine can analyze pre-operative images ofthe eye and provide recommendations for certain aspects of theophthalmic procedure (e.g. recommended positions for incisions) and theGUI can display incision overlays 410, 412 on the image of the eye tohelp guide the surgical staff. Also, in some cases where the ophthalmicprocedure includes a laser treatment component, the GUI can display alaser targeting overlay 414. Also, the surgical suite optimizationengine and/or the heads-up display can receive real-time surgical data(e.g. intraocular pressure) and the GUI can display a real-time surgicaldata overlay 416.

In addition to the main surgical window 402, the GUI 400 can display awide variety of additional GUI features that are helpful to a surgicalteam. For example, the surgical suite can include a diagnostic device(e.g. an Optical Coherence Tomography (OCT) device) and the GUI 400 candisplay a diagnostic imaging window 418 that include a diagnostic viewof eye anatomy. The diagnostic imaging window 418 can displaypre-operative diagnostic images, real-time diagnostic images, modelimages, etc. In some cases, the surgical suite optimization engine canrecognize (e.g. based on surgical procedure data, location of surgicaltools, etc.) a location of the eye that a surgeon is currently focusingon and can cause the diagnostic device to focus on the location. In somecases, a picture-in-picture image 460 (e.g. a picture-in-picturediagnostic image) can be overlaid on the main surgical window 402.

The surgical suite optimization engine can also receive surgicalprocedure data from a surgical console, a surgical guidance application,or other components in the surgical suite and can cause the GUI todisplay the surgical procedure data. For example, the GUI 400 candisplay a current procedure stage 420 and/or current procedural step422.

As explained above, the surgical suite optimization engine can analyzean image of anatomy and digitally process the image to focus on certainanatomy, highlight certain features, etc. In some cases, the surgicalsuite optimization engine can process a variety of distinct surgicalviews that each include various processed data (e.g. a view that focuseson a macula, a view that focuses on a retinal tear, etc.) and the GUI400 can display an interactive stack of surgical views 424. A surgicalteam member can select a particular surgical view from the interactivestack of surgical views 424 and the selected view can be displayed inthe main surgical view window 402.

The GUI 400 can also include a variety of menu items 426, 428, 430, 432,434, 436 for interacting with the surgical suite optimization engine orother components in the surgical suite. For example, the GUI 400 caninclude a surgical plan menu item 426 for interacting with a surgicalplanning application, surgical plan preferences, etc. The GUI 400 canalso include a surgical practice management application 428, a technicalsupport contact menu 430, etc. Also, the GUI 400 can include controlmenus such as a supplemental optics/overlay menu 432, a display positionmenu 434, a diagnostic device control menu 436, etc.

The GUI 400 can also include a variety of icons for controlling aspectsof the display such as an Information Icon 438, a color adjustment icon440, a camera adjustment icon 442, a monitor adjustment 444, a screenrecording 446, an input source icon 448, a screen layout icon 450, asurgical step icon 452, etc.

A wide variety of common approaches can be employed for interacting withthe GUI 400 displayed on a heads-up display, e.g. a mouse, a touchpad,etc. Additionally, the heads-up display can include a touchscreen andthe GUI 400 can be interacted with via the touchscreen. However,maintaining a sterile environment can be important in a surgical suiteand a surgeon may already be using both hands for manipulating tools andhis feet with foot pedals. Therefore, contact-less interaction with theGUI 400 can be helpful. As explained above, in some cases, the surgicalsuite optimization engine includes a voice control module forrecognizing voice commands for interacting with the GUI 400.

Additionally, in some cases, the surgical suite optimization engine canemploy gaze-tracking to interact with the GUI 400. For example, surgicalglasses used to view a stereoscopic image can include an eye trackingmechanism (e.g. a camera, one or more accelerometer etc.) that candetect eye position, movement of gaze, duration of focus, etc. and theGUI 400 can be interacted with using the detected eye/gaze position.

Some embodiments of gaze-tracking involve placing two or moreaccelerometers into the lightweight glass frames. These glasses canfacilitate positioning of a digital pointer to on-screen menus andactivating a function of the menu after the pointer remains on the menufor a threshold period of time or after verifying selection with asecond selection function (e.g., foot pedal, finger embedded with sensorselection technology, voice, etc.). The digital pointer allows selectionassistance via x or y-axis movements for navigating through menus andselecting preferred image or options. In some cases, the accelerometersdetermine when surgeon's gaze remains on a particular part of the GUI400 for a threshold time (e.g. two seconds) and cause a pointer toappear on the GUI 400. In some cases, once the pointer is displayed, amenu will pop up to confirm the selection.

FIG. 4B illustrates a method 470 of gaze-enabled control of a graphicaluser interface. The method involves configuring three-dimensionalglasses with accelerometers 472 transmitting accelerometer data tosurgical suite optimization engine 474, and determining gaze positionbased on accelerometer data 476. Next, the method 470 involvesdetermining when gaze remains fixed for threshold period 478 (e.g. twoseconds). When the threshold is not reached, the method 470 continues todetermining gaze position based on accelerometer data 476. When thethreshold gaze fixture is reached, the method 470 involves determininggaze position based on accelerometer data and displaying a pointer thatfollows the gaze 480. Next, the method 470 involves determining when thepointer remains fixed on menu item for an additional threshold period(e.g. two seconds, three seconds) 482. When gaze fixture on the menuitem does not reach the additional threshold, the method 470 continuesto determining gaze position based on accelerometer data 476. When gazefixture on the menu item reaches the additional threshold, the method470 involves activating the menu item 484 and fading out pointer after afurther threshold period 486.

The GUI 400 described herein is one example of a wide variety ofinterfaces that can be used for conducting surgical procedures. Thebenefits of such a GUI is even increased by the surgical suiteoptimization engine. As explained above, in response to receivingsignals from other surgical components, the surgical suite optimizationengine can perform a variety of actions to optimize the ophthalmicsurgery suite.

Adjusting Display Settings

An example of the surgical suite optimization engine performing anaction to optimize the ophthalmic surgery suite involves a surgicalconsole transmitting procedure data and the surgical suite optimizationengine adjusting display settings in the heads-up display based on theprocedure data.

FIG. 5A illustrates a method 500 of adjusting display settings based onsurgical progression. The method 500 involves arranging a surgical suitewith a surgical camera, a surgical console, a movable heads-up display,and a surgical suite optimization engine 505. Next, the method 500involves receiving, from the surgical camera, a three-dimensional imageof an eye 510 and displaying a stereoscopic representation of athree-dimensional image of an eye received from the surgical camera 515.

The surgical suite optimization engine can be communicatively coupledwith the surgical console and the method 500 can involve the surgicalsuite optimization engine receiving surgical procedure data 520 from thesurgical console. To enhance the display, the method 500 can involvedetermining, based on procedure data, current surgical step 525 andadjusting display settings based on surgical step 530. Next, the method500 can involve detecting progression to next surgical step 535 andadjusting display settings based on surgical progression 540. FIGS.5B-5E illustrate examples of display settings for various steps of anophthalmic procedure.

Optimizing Stereopsis

In some cases, the surgical suite optimization engine can receiveinformation about steps/stages in a surgical procedure and can optimizestereopsis for surgical staff viewing the displayed stereoscopicrepresentation of a three-dimensional image of eye anatomy in responseto a change in the phase of a surgical procedure and/or procedure type,a manual command, a user preference, etc.

Using a HDR camera enables a high quality, three-dimensional image ofpatient anatomy. Also, a stereoscopic effect and/or hypersteropsis canbe achieved to provide the surgical staff to perceive relative depth inthe display of the patient anatomy. These effects are a function ofseveral parameters including distance between cameras and the distanceof the observing surgeon's eyes in relation to the display. Also,individual surgeons may prefer more or less stereopsis. Individualsurgeons may also want the stereopsis modified for different proceduretypes or different surgical steps within a procedure type. In someembodiments of the present technology, a heads-up display can be mountedto a movable mechanism that moves the heads-up display to dynamicallyadjust the distance of the observing surgeon's eyes in relation to thedisplay.

FIGS. 6A and 6B illustrate a system 600 for adjusting the distance of aheads-up display 604 in relation to an observation point according tosome embodiments of the present technology. The system 600 includes asurgical camera 608 positioned above a patient table 610. The surgicalcamera 608 is configured to capture a three-dimensional image of patientanatomy for a patient on the patient table 610 and transmit the imagedata to a heads-up display 604. A surgeon can be positioned on a stool612, for example, near the head of the patient and perform a procedureon the patient using a surgical console (not shown).

As shown, the heads-up display 604 is floor-mounted with a mount base630 and a support member 632. The mount base 630 can include anadjustment mechanism for moving the support member in a z-direction tochange the distance between the heads-up display 604 and the surgeon.For example, the mount base 630 can employ electro-mechanic solutionssuch as linear actuators, screw-drive mechanisms as well as hydraulics,pneumatics, etc. Also, the support member 632 can be adjusted in the x-and y-directions. For example, the support member 632 can include avertical telescopic member that can be adjusted in the y-direction and ahorizontal adjustment member (not shown) attached to the back of theheads-up display 604 that can be adjusted in the x-direction.

Although a floor-mounted system is illustrated, those with ordinaryskill in the art having the benefit of the present disclosure willreadily appreciate that a wide variety of mounting and adjustmentsystems can be employed to achieve the stereoscopic optimization benefitdescribed herein. For example, a ceiling mount may have advantages basedon a greater range of unobstructed movement compared to a floor mountedconfiguration. Also, although an ophthalmic surgery system is described,those with ordinary skill in the art having the benefit of the presentdisclosure will readily appreciate that a wide variety of proceduretypes can benefit from the stereoscopic optimization described herein.For example, otorhinolaryngology procedures, neurosurgeries, necksurgeries, back surgeries, internal chest surgeries, joint surgeries,ligament surgeries, muscle surgeries, bone surgeries, organ surgeries,and dental surgeries would particularly benefit.

In some cases, the heads-up display 604 can present an interactivegraphical user interface (GUI) with interactive elements for allowing auser to configure a display position program. For example, the displayposition program can allow the surgical staff to manually adjust theposition of the display relative to the surgeon, enter surgeonpreferences, enable adjustment of the display distance based on a typeof procedure, a step or phase in the procedure, etc. The heads-updisplay 604 can also include a surgical suite optimization engine (notshown) which can receive surgical procedure data (e.g. from the surgicalconsole) that describes steps and/or phases of a particular procedure.For example, in cataract surgeries, the surgical procedure data candescribe a capsulorhexis step, a phacoemulsification step, anirrigation/aspirating step, etc. and in retinal surgeries, the proceduredata can describe an ILM peel step, a vitrectomy step, a Scleral bucklestep, etc. Likewise, in a combination anterior/posterior procedure, theprocedure data can describe a Cataract Phase and a Retinal Phase. Thesurgical suite optimization engine can also include a database ofoptimal display positions based on surgical steps/phases and a databasethat stores surgeon preferences.

The surgical suite optimization engine can receive surgical proceduredata, monitor surgical progression (e.g. via feedback from the surgicalconsole), detect one or more changes in surgical phase and/or surgicalstep, determine an optimal distance, and perform the action of movingthe heads-up display 604 based on procedure phase/step and surgeonpreferences. For example, moving the heads-up display 604 can involvethe Surgical Suite Optimization engine transmitting an instruction thatcauses one or more actuator to move the heads-up display.

In FIG. 6A, the heads-up display 604 is located a distance D1 away fromthe position of the surgeon. FIG. 6B illustrates the system 600 afterthe surgical suite optimization engine detects a change in surgicalstep/phase and moves the heads-up display 604 to a distance D2 away fromthe position of the surgeon to optimize the stereoscopic effect for thecurrent step/phase of the procedure.

FIG. 6C illustrates a method 650 of optimizing stereopsis for surgicalstaff viewing the displayed stereoscopic representation of athree-dimensional image of eye anatomy in response to a change in thephase of a surgical procedure and/or procedure type. Initially, themethod 650 involves arranging a surgical suite with a surgical camera, asurgical console, a movable heads-up display, and a surgical suiteoptimization engine 652.

Next, the method 650 involves receiving, from the surgical camera, athree-dimensional image of an eye 654 and displaying a stereoscopicrepresentation of a three-dimensional image of an eye received from thesurgical camera 656 on the heads-up display. The method 650 proceeds bythe surgical suite optimization engine receiving surgical procedure data658. For example, a surgical console can transmit surgical proceduredata to the surgical suite optimization engine. Also, in some cases,surgical procedure data can be generated by one or more surgicalguidance applications, a pre-operative surgical planning application,etc.

In some cases, a method 650 of optimizing stereopsis can involve thesurgical suite optimization engine determining an initial defaultposition for the heads-up display based on the procedure data 660, e.g.by referencing a database describing optimal distances based onprocedure, analyzing historical surgical data describing best practices,etc. Next, the method 650 can involve the surgical suite optimizationengine transmitting an instruction to a display positioning system (e.g.movable base, actuating arm, etc.) and moving the heads-up display todefault position based on procedure data 662.

Next, the method 650 involves accessing surgeon preferences from profiledatabase and/or receiving manual position setting via a GUI 664,adjusting display orientation settings based on surgeon preferencesand/or manual selections 666, and monitoring surgical suite for actioncode relating to surgical progression 668. In some cases, a surgicalconsole communicatively coupled with the surgical suite optimizationengine transmits an action code (or other signal) when a change in thesurgical step/phase, e.g. automatically when a new instrument is used,after receiving manual input from surgical staff, after recognizing aspeech command from the surgical staff, etc. The method 650 involvesdetecting an action code describing a change in surgical step/phase 670,determining an optimal distance for the heads-up display for newsurgical phase to optimize stereopsis 672, and moving heads-up displayto optimal distance 674.

Image Auto-Centration and Auto-Coaxial Illumination for Optimal RedReflux

Another example of the surgical suite optimization engine performing anaction to optimize the ophthalmic surgery suite involves automaticallycentering surgical camera with a patient's eye. In a three-dimensionalsurgery system, the surgical camera should remain centered on the pupilsuch that the eye can be presented centrally and fully on the heads-updisplay without vignettes and to ensure optimal lighting conditions,i.e. Red Reflex. However, as a patient's head and/or eye moves or isrepositioned by the surgeon, the patient's eye can drift outside of thesurgical camera's image center causing surgical staff to reposition thepatient or the camera.

Some embodiments of the present technology involve the surgical cameramounted on a positioning mechanism and involve the surgical cameraand/or the surgical suite optimization engine configured to track aposition of the eye (e.g. using anatomical registration). Thepositioning mechanism can be, for example, a robotic arm, a gimbal, alinear positioning system, etc. Further, the surgical camera and/or thesurgical suite optimization engine can include control system that cancontrol the positioning mechanism to reposition the surgical camera toremain centered on the eye to ensure centration on the viewing displayand to ensure that light source(s) to always be substantiallyperpendicular to the pupil. Tracking the eye and controlling thepositioning mechanism can ensure centration and bright illuminationconditions, e.g. optimized red reflex.

Further, in some embodiments, color and light post-processing imagingalgorithms may also be employed by the surgical suite optimizationengine to optimize red reflex images. Also, some specific lightwavelengths can be selected/filtered that better penetrate mediaopacities of the cornea, crystalline lens, or vitreous that may bedetrimental to establishing an optimized red reflex. Similarly, adaptiveoptics may help eliminate aberrations that can diminish a bright redreflex and auto-focus in one or multiple zones.

FIG. 7 illustrates a method 700 of ensuring centration of a surgicalcamera on an eye according to some embodiments of the presenttechnology. The method 700 involves arranging a surgical suite with aheads-up display, a surgical camera having a positioning system andadaptive optics, and a surgical suite optimization engine 705. In somecases, adaptive optics are selected to eliminate aberrations in thecornea, crystalline lens, vitreous, etc. that may inhibit optimalimaging of the anterior surfaces of the eye such as the anterior capsulesurface, posterior capsule surface, etc. Elimination of the opacitiescombined with image centration can result in optimal light conditionsand anatomical clarity. The method 700 can further include selectinglight characteristics for the surgical camera to optimize imaging of eyeanatomy 710. For example, selecting light characteristics can includeselecting wavelengths that better penetration of media opacities of thecornea, crystalline lens, vitreous, etc., selecting wavelengths that aresafer to the human eye, eliminating particular region (e.g. blue) of thespectrum, etc.

Next, the method 700 can involve a surgical suite optimization enginereceiving, from the surgical camera, a three-dimensional image of an eye715 and displaying, on the heads-up display, a stereoscopicrepresentation of a three-dimensional image of an eye received from thesurgical camera 720. The method 700 also involves the surgical suiteoptimization engine registering anatomical features of eye 725 andtracking, using registered anatomical features of eye, the position ofthe eye 730. As the surgical suite optimization engine tracks theposition of the eye, the method 700 can involve the surgical suiteoptimization engine generating, in substantially real-time, a controlsignal for controlling the positioning system attached to the surgicalcamera 735 and transmitting control signal for re-positioning thesurgical camera to ensure centration on the eye 740.

Adjusting Diagnostic Device

Another example of the surgical suite optimization engine performing anaction to optimize the ophthalmic surgery suite involves the surgicalsuite optimization engine enhancing diagnostic visualization of anatomyand pathology. Many of the anatomical aspects of the eye aretransparent, thus playing a role in passing light through the human eyeoptical system to the neuro-sensory retina. To distinguish these tissuesfrom similarly transparent membrane pathologies is challenging evenduring pristine conditions. These clear tissues often become opacifiedor hazy including the cornea, crystalline human lens and capsule, andvitreous opacities and hemorrhage. These sub optimal optical conditionsin inherently clear structures make surgery challenging and limitoptimal surgical precision and subsequent patient outcomes. To optimizevisualization of eye anatomy and pathology, some embodiments of thepresent technology involve digital diagnostic and supplementary imagesthat can help distinguish anatomical and pathological components.

For example, in some cases the Surgical Suite Optimization engine canreceive imagery from both a surgical camera and a diagnostic device,e.g. an Optical Coherence Tomography (OCT) device; digital endoscopiccameras; laser speckle flowgraphy devices, ultrasound/echograph devices,wavefront and laser devices, light interrogation/reflectance devices,etc. The Surgical Suite Optimization engine can combine the imagery,blend diagnostic views with camera views, etc. Further, the SurgicalSuite Optimization engine can apply digital enhancement technologies tosubtract aberrations from the optical system. For example, in some casesthe Surgical Suite Optimization engine can apply adaptive opticstechnology to automatically strip aberrations and sharpen the clarity ofthe view for the surgeon.

A wide variety of application for diagnostics and adaptive optics can beapplied to enhance a surgical procedure; however, an illustrativeexample involves corneal inlay surgery for addressing presbyopia.Presbyopia affects more patients than any other age related diseaseprocess. A variety of refractive and/or mechanical polymers have beendesigned to be placed intra-cornea to adjust for refractive error and/orpresbyopia. The inlay performance and ease of installation are afunction of imaging and diagnostics available to the surgeon. Thesurgical camera, heads-up display, and surgical suite optimizationengine pf the present technology provides surgeons with an excellentplatform for optical and digital guidance for minimally invasive CornealRefractive/Presbyopia correction. In some cases the surgical suiteoptimization engine can define diagnostic settings and perform digitalimage post processing with color attribute filtering and/or imageenhancement technology such as adaptive optics and can make the inlayprocedure more precise for the ophthalmic surgeon. These technologiescan provide complementary images to the traditional optical view toprovide more informed decision making for the surgeon.

FIG. 8 illustrates a method 800 of enhancing diagnostic visualizationfor surgical staff viewing a stereoscopic representation of athree-dimensional image of eye anatomy. Initially, the method 800involves arranging a surgical suite with a heads-up display with asurgical suite optimization engine, a surgical camera, a surgicalconsole, and a diagnostic device (e.g. an Optical Coherence Tomography(OCT) device; digital endoscopic cameras; laser speckle flowgraphydevices, ultrasound/echograph devices, wavefront and laser devices,light interrogation/reflectance devices, etc.) 805.

Next, the method 800 involves receiving, from the surgical camera, athree-dimensional image of an eye 810 and displaying a stereoscopicrepresentation of a three-dimensional image of an eye and at least onediagnostic view 815 on the heads-up display. The method 800 proceeds bythe surgical suite optimization engine receiving surgical procedure data820. For example, a surgical console can transmit surgical proceduredata to the surgical suite optimization engine. Also, in some cases,surgical procedure data can be generated by one or more surgicalguidance applications, a pre-operative surgical planning application,etc.

Next, the method 800 involves selecting initial diagnostic settings andsupplemental optics based on surgical procedure data 825, accessingsurgeon preferences from profile database 830, adjusting diagnosticsetting and supplemental optics based on surgeon preferences 835, andmonitoring surgical suite for action code relating to surgicalprogression 840. In some cases, a surgical console communicativelycoupled with the surgical suite optimization engine transmits an actioncode (or other signal) when a change in the surgical step/phase, e.g.automatically when a new instrument is used, after receiving manualinput from surgical staff, after recognizing a speech command from thesurgical staff, etc. The method 800 involves detecting an action codedescribing a change in surgical step/phase 845 and adjusting diagnosticsettings and supplemental optics based on change in surgical step andsurgeon preferences 850. In some cases, adjusting diagnostic settingscan involve moving the diagnostic device (e.g. on a robotic arm) to aimthe diagnostic device onto anatomy relevant to a surgical step/phase.

Dynamically Adjusting Anatomical Area of Focus, Anatomical Highlighting,and Graphical Overlays

Another example of the surgical suite optimization engine performing anaction to optimize the ophthalmic surgery suite involves dynamicallyadjusting anatomical area of focus, anatomical highlighting, andgraphical overlays on or near the three-dimensional stereoscopicrepresentation of patient's eye or in another location in the GUIdisplayed on the heads-up display.

The surgical suite optimization engine can be used with a surgicalcamera and heads up display to perform a large variety of ophthalmicprocedures. Over the course of these procedures, it is beneficial forthe surgical staff to observe a wide variety of areas of focus andspecific anatomical features. In some embodiments of the presenttechnology, the surgical suite optimization engine can monitor and/oranticipate surgical progression and can automatically focus on desiredareas of focus and automatically highlight specific anatomical featuresby applying a variety of rules. Also wide variety of overlays can beused to optimize surgery. For example, pre-operative diagnostic imagesand planning tools can be overlaid on a surgical image of an eye toguide a surgeon in the placement of incisions, the alignment of aninter-ocular lens, etc. The surgical suite optimization engine of thepresent technology can also determine to adjust overlays by monitoringsurgical progression and by applying rules for adjusting overlays.

In some embodiments of the present technology, the surgical suiteoptimization engine can apply pre-programmed rules for adjusting area offocus, anatomical highlighting, and/or graphical overlays. For example,the surgical suite optimization engine can receive surgical progressiondata from a surgical console and consult a display options database toaccess pre-programmed rules. In some cases, the Surgical SuiteOptimization engine can also reference one or more surgical practiceguidance application and access pre-programmed overlay templatepackages, e.g. used optimal by well-respected surgeons. Also, thesurgical suite optimization engine can apply machine learning andartificial intelligence techniques and/or provide surgical data to aseparate machine learning and artificial intelligence engine and requesttemplates for dynamically adjusting anatomical area of focus, anatomicalhighlighting, and graphical overlays. For example, in some cases, theSurgical Suite Optimization engine can receive pre-programmed area offocus, highlighting and overlay templates created through machinelearning and/or artificial intelligence that are determined to result inoptimal patient outcomes.

FIG. 9 illustrates a method 900 of a surgical suite optimization enginedynamically adjusting anatomical area of focus, anatomical highlighting,and graphical overlays according to some embodiments of the presenttechnology. The method 900 involves arranging an improved surgical suitewith a heads-up display, a surgical suite optimization engine, asurgical camera, and a surgical console 905. Next, the method 900involves receiving, from the surgical camera, a three-dimensional imageof an eye 910 and displaying a stereoscopic representation of athree-dimensional image of an eye received from the surgical camera 915.Further, the method 900 can involve receiving surgical plan 920. In somecases, a surgical plan can be received from a surgical console, asurgical guidance application, a machine learning and artificialintelligence engine, etc.

The method 900 further involves the surgical suite optimization engineselecting initial area of focus, anatomical highlighting, and/orgraphical overlays 925, monitoring a surgical progression 930, detectinga change in surgical step/phase 935 and adjusting area of focus,anatomical highlighting, and/or graphical overlays 940 based on the newsurgical step/phase.

Providing Surgical Alerts and Recommendations

Another example of the surgical suite optimization engine performing anaction to optimize the ophthalmic surgery suite involves monitoringcontrol limits and providing surgical alerts and recommendations.

In some cases, a surgical console (e.g. a surgical console configuredfor performing vitreoretinal surgery) can warn or prevent the user whencertain thresholds are exceed to avoid potentially unsafe operations.For example, the use of low vacuum limits, low flow rates, lowirrigation pressure, high power settings, extended power usage, powerusage during occlusion conditions, failure to sufficiently aspirateviscoelastic prior to using power, excessively tight incisions, andcombinations of the above actions may result in significant temperatureincreases at incision site and inside the eye, and lead to severethermal eye tissue damage. A surgical console can enforce control limitsto warn or prevent these and other potentially adverse conditions.

FIG. 10 illustrates a method 1000 of monitoring control limits andproviding surgical alerts and recommendations. The method 1000 involvesarranging an improved surgical suite with a heads-up display, a surgicalsuite optimization engine, a surgical camera, and a surgical console1005, receiving, from the surgical camera, a three-dimensional image ofan eye 1010, and displaying a stereoscopic representation of athree-dimensional image of an eye received from the surgical camera1015. In some cases, the surgical console can transmit surgicalprocedure data to the surgical suite optimization engine that includesone or more control limit threshold settings, e.g. a maximum advisableinter-ocular pressure. In some other embodiments, an additional surgicalpractice guidance application can include one or more sources of controllimit settings. For example, a surgical practice guidance applicationcan include settings defined as optimal by well-respected surgeons,settings determined through machine learning and/or artificialintelligence as resulting in optimal patient outcomes, etc.

Next, the method 1000 involves receiving surgical procedure dataincluding one or more control limits for aspect(s) of the procedure 1020and monitoring surgical progression using feedback from surgicalconsole, accessories, etc. 1025. For example, monitoring surgicalprogression can include monitoring intraocular pressure of a patient'seye by the surgical console. The method 1000 then involves detectingwhen an aspect of procedure comes within a predetermined thresholdproximity to a control limit 1030 and causing the heads-up display todisplay an alert and/or recommendation on the heads-up display 1035.

Color Modulation for Laser Operation

Another example of the surgical suite optimization engine performing anaction to optimize the ophthalmic surgery suite involves monitoringcontrol limits and providing surgical alerts and recommendations. Poorvisualization during surgical procedures can deter optimal surgicaltasks and subsequently, limit optimal patient outcomes. However, thesurgical suite optimization engine of the present technology can be usedto adjust post-image capture image characteristics including colorcharacteristics (e.g. saturation, contrast, gain, gamma, cast etc.) andlight characteristics (e.g. exposure, contrast, blackpoint etc.) Forexample, while using a laser in retina surgery, there is typically a redaiming beam ˜635 nm wavelength and a subsequent 532 nm green treatmentor photocoagulation beam. However, the red aiming beam can be difficultto see on the orange background of the human retina. This makes itchallenging for the surgeon to optimize precise targeting of thetreatment beam. Also, the treatment beam can create a “green flashback”as the beam strikes the targeted retina and may temporarily obscure thesurgeon's vision and can compromise dark adaptation. Modulating colorand/or light characteristics may be helpful to the surgeon to optimizevisualization and patient outcomes. Accordingly, the surgical suiteoptimization engine can modulate pixel colors post-image capture. Forexample, the surgical suite optimization engine can accentuate the redaiming beam by adjusting specific tertiary color spectra to maximize redcontrast. Modifying portions of the green spectrum opposite of 635 nmred (e.g. in Itten's color contrast circle) can maximize red contrast inthe displayed image. Similarly, during green laser flashback thesurgical suite optimization engine can neutralize green by temporarilyreducing or eliminating the capability to display green. The surgicalsuite optimization engine can also truncate or modulate broad sectionsof spectra of primary (RGB) or secondary spectra (OPG) colors. Linkingthese color modulation techniques utilizing a graphics processingcomputer (GPU) of the surgical suite optimization engine provides avaluable integrated approach to optimizing surgeon performance.

In a specific example, when the Laser system software (e.g. a laseritself, a wireless footswitch coupled with the laser, the surgicalconsole, etc.) communicates with the surgical suite optimization engine,the image modulations are coordinated. The state of the laser wouldinvoke different modulation effects: in a first state (“Laser ready”),the surgical suite optimization engine optimizes the red aiming beamintensity and contrast; and in a second state (“Laser Firing”), thesurgical suite optimization engine modulates or eliminates the greenflashback. The surgical suite optimization engine can alternate theseviews instantaneously as the Laser is “aimed”, then “fired”, then“aimed” at a subsequent target and so on until all Laser spots areapplied.

FIG. 11A illustrates a method 1100 of modulating color characteristicsof a stereoscopic representation of a three-dimensional image of eyeanatomy to optimize visualization during a laser treatment step of anophthalmic procedure. The method 1100 can involve arranging an improvedsurgical suite with a heads-up display, a surgical camera and a surgicalconsole having a laser treatment component 1105, receiving, from thesurgical camera, a three-dimensional image of an eye 1110, anddisplaying a stereoscopic representation of a three-dimensional image ofan eye received from the surgical camera 1115. Next, the method 1100 caninvolve receiving surgical procedure data including laser state data1120. For example, the laser state data can involve data indicatingwhether a laser used for photocoagulation of tissue in a retinalprocedure is in an “aiming” state or a “firing” state. The laser statedata can be transmitted from the surgical console, a laser modulecoupled to the surgical console, a stand-alone laser module, a wirelessfootswitch communicatively coupled with a laser module and configured toalternatively aim and fire the laser, etc.

Next, the method 1100 can involve modulating color characteristics tooptimize visualization during laser treatment 1125. For example,modulating color characteristics to optimize visualization during lasertreatment can involve performing digital signal processing to maximizered contract of the stereoscopic representation of the three-dimensionalimage of the eye when the laser treatment transitions from the emissionof the treatment beam to the emission of the aiming beam andneutralizing a green flashback from a retina in the stereoscopicrepresentation of the three-dimensional image of the eye when the lasertreatment transitions from the emission of the aiming beam to theemission of the treatment beam. In some cases, the surgical suiteoptimization engine can include a dedicated graphical processing unitfor performing digital signal processing.

FIGS. 11B and 11C illustrate examples of graphical user interfacesshowing optimized visualization for two laser states according to someembodiments of the present technology. As represented in FIGS. 11B and11C, a graphical user interface (GUI) 1130 of a display dashboardincludes a main surgical view window 1130 that displays a stereoscopicrepresentation of a three-dimensional image of an eye received from thesurgical camera. In FIG. 11B, the image of the eye is modulated towardred to maximize red contract during a laser aiming step. In FIG. 11C,the image of the eye is modulated toward green to neutralize greenflashback during a laser firing step.

Automatically Contacting Support

Another example of the surgical suite optimization engine performing anaction to optimize the ophthalmic surgery suite involves identifying aproblem in the surgical suite and automatically initiating contact witha support professional to remedy the problem. One or more devices,accessories, software modules, etc. connected in a surgical suite canreport errors to the surgical suite optimization engine and the surgicalsuite optimization engine can perform actions in response to the errors.For example, a surgical console can issue an error code in the eventthat one or more of the surgical console's components and/or softwarefunctions malfunctions or does not execute properly. In response toreceiving the error code, the surgical suite can transcode the actioncode to a known format, if necessary, and determine to perform an actionin response to the malfunction described in the action code. Forexample, when a component of a surgical console malfunctions, thesurgical suite optimization engine can automatically initiate a call orvideo conference with a technical support team responsible for providingsupport for the surgical console. In another example, the surgicalconsole can transmit an action code when a component is nearing itswarranty end of life and the surgical suite optimization engine anautomatically schedule a call with a sales representative responsiblefor maintaining a surgery practice's account. Those with ordinary skillin the art having the benefit of the present disclosure will readilyappreciate that a wide variety actions can be supported by the surgicalsuite optimization engine in response to receiving error codes fromcomponents in the surgical suite.

FIG. 12 illustrates a method 1200 of identifying a problem in thesurgical suite and automatically initiating contact with a supportprofessional to remedy the problem. The method 1200 involves arrangingan improved surgical suite with a heads-up display, a surgical cameraand a surgical console 1205, receiving, from the surgical camera, athree-dimensional image of an eye 1210, and displaying a stereoscopicrepresentation of a three-dimensional image of an eye received from asurgical camera 1215. Next, the method 1200 can involve receiving, fromthe surgical console, an action code describing a malfunction of theoperational status of the surgical console 1220, transcoding the actioncode and determining to perform an action 1225, and initiating a videoconference with a technical support center 1230. Next, the method 1200can involve re-arranging display elements to include the stereoscopicrepresentation of the three-dimensional image of the eye along with avideo conference GUI 1235 and presenting, on the display, the videoconference in the video conference 1240.

Automatic Inventory Management

Another example of the surgical suite optimization engine performing anaction to optimize the ophthalmic surgery suite involves determiningwhen an inventory level is low and automatically ordering new inventory.In some surgical procedures, consumable accessories are used to ensuresterility. Consumable accessories can include a tag (e.g. RFID tag) orother means to identify the consumable accessory in order for thesurgical console to determine that the right surgical accessory is beingused for the procedure and for the particular patient. Also, the tag canbe used to manage inventory of consumable accessories, e.g. in aninventory management system. In some embodiments of the presenttechnology, the surgical suite optimization engine can determine whenparticular consumable accessories are used (e.g. when they are scannedwhen being connected with a surgical console) and can automaticallyre-order inventory when a level of the consumable accessory fallsbeneath a predetermined threshold level of inventory.

FIG. 13 illustrates a method 1300 of a surgical suite optimizationengine maintaining appropriate inventory for a surgical practice. Themethod 1300 involves arranging an improved surgical suite with asurgical console, a surgical suite optimization engine and an inventorymanagement application 1305. Next, the method 1300 involves determiningthe consumption of a consumable surgical accessory 1310 (e.g. by asurgical console reading an RFID tag of a consumable accessory), andtransmitting an action code indicating the consumption of the consumablesurgical accessory 1315.

Next, the method 1300 involves a surgical suite optimization enginedetermining auto-order preferences for surgical practice 1320 (e.g. byaccessing preferences for a surgical practice stored in a database inthe surgical suite optimization engine) and determining that a level ofthe surgical consumable accessory is beneath a pre-determined thresholdlevel 1325, as defined in the auto-order preferences. Next, the method1300 involves the surgical suite optimization engine automaticallyordering additional inventory for the consumable surgical accessory1330, as defined in the auto-order preferences.

Example System Architecture

FIG. 14A and FIG. 14B illustrate possible system embodiments. The moreappropriate embodiment will be apparent to those of ordinary skill inthe art when practicing the present technology. Persons of ordinaryskill in the art will also readily appreciate that other systemembodiments are possible.

FIG. 14A illustrates a conventional system bus computing systemarchitecture 1400 wherein the components of the system are in electricalcommunication with each other using a bus 1405. Exemplary system 1400includes a processing unit (CPU or processor) 1410 and a system bus 1405that couples various system components including the system memory 1415,such as read only memory (ROM) 1420 and random access memory (RAM) 1425,to the processor 1410. The system 1400 can include a cache of high-speedmemory connected directly with, in close proximity to, or integrated aspart of the processor 1410. The system 1400 can copy data from thememory 1415 and/or the storage device 1430 to the cache 1412 for quickaccess by the processor 1410. In this way, the cache can provide aperformance boost that avoids processor 1410 delays while waiting fordata. These and other modules can control or be configured to controlthe processor 1410 to perform various actions. Other system memory 1415may be available for use as well. The memory 1415 can include multipledifferent types of memory with different performance characteristics.The processor 1410 can include any general purpose processor and ahardware module or software module, such as module 1 1432, module 21434, and module 3 1436 stored in storage device 1430, configured tocontrol the processor 1410 as well as a special-purpose processor wheresoftware instructions are incorporated into the actual processor design.The processor 1410 may essentially be a completely self-containedcomputing system, containing multiple cores or processors, a bus, memorycontroller, cache, etc. A multi-core processor may be symmetric orasymmetric.

To enable user interaction with the computing device 1400, an inputdevice 1445 can represent any number of input mechanisms, such as amicrophone for speech, a touch-sensitive screen for gesture or graphicalinput, keyboard, mouse, motion input, speech and so forth. An outputdevice 1435 can also be one or more of a number of output mechanismsknown to those of skill in the art. In some instances, multimodalsystems can enable a user to provide multiple types of input tocommunicate with the computing device 1400. The communications interface1440 can generally govern and manage the user input and system output.There is no restriction on operating on any particular hardwarearrangement and therefore the basic features here may easily besubstituted for improved hardware or firmware arrangements as they aredeveloped.

Storage device 1430 is a non-volatile memory and can be a hard disk orother types of computer readable media which can store data that areaccessible by a computer, such as magnetic cassettes, flash memorycards, solid state memory devices, digital versatile disks, cartridges,random access memories (RAMs) 1425, read only memory (ROM) 1420, andhybrids thereof.

The storage device 1430 can include software modules 1432, 1434, 1436for controlling the processor 1410. Other hardware or software modulesare contemplated. The storage device 1430 can be connected to the systembus 1405. In one aspect, a hardware module that performs a particularfunction can include the software component stored in acomputer-readable medium in connection with the necessary hardwarecomponents, such as the processor 1410, bus 1405, display 1435, and soforth, to carry out the function.

FIG. 14B illustrates a computer system 1450 having a chipsetarchitecture that can be used in executing the described method andgenerating and displaying a graphical user interface (GUI). Computersystem 1450 is an example of computer hardware, software, and firmwarethat can be used to implement the disclosed technology. System 1450 caninclude a processor 1455, representative of any number of physicallyand/or logically distinct resources capable of executing software,firmware, and hardware configured to perform identified computations.Processor 1455 can communicate with a chipset 1460 that can controlinput to and output from processor 855. In this example, chipset 860outputs information to output 1465, such as a display, and can read andwrite information to storage device 1470, which can include magneticmedia, and solid state media, for example. Chipset 1460 can also readdata from and write data to RAM 1475. A bridge 1480 for interfacing witha variety of user interface components 1485 can be provided forinterfacing with chipset 1460. Such user interface components 1485 caninclude a keyboard, a microphone, touch detection and processingcircuitry, a pointing device, such as a mouse, and so on. In general,inputs to system 1450 can come from any of a variety of sources, machinegenerated and/or human generated.

Chipset 1460 can also interface with one or more communicationinterfaces 1490 that can have different physical interfaces. Suchcommunication interfaces can include interfaces for wired and wirelesslocal area networks, for broadband wireless networks, as well aspersonal area networks. Some applications of the methods for generating,displaying, and using the GUI disclosed herein can include receivingordered datasets over the physical interface or be generated by themachine itself by processor 1455 analyzing data stored in storage 1470or 1475. Further, the machine can receive inputs from a user via userinterface components 1485 and execute appropriate functions, such asbrowsing functions by interpreting these inputs using processor 1455.

It can be appreciated that exemplary systems 1400 and 1450 can have morethan one processor 1410 or be part of a group or cluster of computingdevices networked together to provide greater processing capability.

For clarity of explanation, in some instances the present technology maybe presented as including individual functional blocks includingfunctional blocks comprising devices, device components, steps orroutines in a method embodied in software, or combinations of hardwareand software.

In some embodiments the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Methods according to the above-described examples can be implementedusing computer-executable instructions that are stored or otherwiseavailable from computer readable media. Such instructions can comprise,for example, instructions and data which cause or otherwise configure ageneral purpose computer, special purpose computer, or special purposeprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The computer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware, orsource code. Examples of computer-readable media that may be used tostore instructions, information used, and/or information created duringmethods according to described examples include magnetic or opticaldisks, flash memory, USB devices provided with non-volatile memory,networked storage devices, and so on.

Devices implementing methods according to these disclosures can comprisehardware, firmware and/or software, and can take any of a variety ofform factors. Typical examples of such form factors include laptops,smart phones, small form factor personal computers, personal digitalassistants, and so on. Functionality described herein also can beembodied in peripherals or add-in cards. Such functionality can also beimplemented on a circuit board among different chips or differentprocesses executing in a single device, by way of further example.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are means for providing the functions described inthese disclosures.

The above disclosed subject matter is to be considered illustrative, andnot restrictive, and the appended claims are intended to cover all suchmodifications, enhancements, and other embodiments which fall within thetrue spirit and scope of the present disclosure. Thus, to the maximumextent allowed by law, the scope of the present disclosure is to bedetermined by the broadest permissible interpretation of the followingclaims and their equivalents, and shall not be restricted or limited bythe foregoing detailed description.

1-41. (canceled)
 42. An ophthalmic system comprising: a surgical consolehaving tools for conducting an ophthalmic procedure, a consoleprocessor, and a console memory containing console instructions which,when executed by the console processor, cause the console processor toaccess procedure data stored in the console memory, the procedure datadescribing a plurality of procedure stages and determine when totransmit an action code relating to a transition between a firstprocedure stage and a subsequent procedure stage; a surgical camera forcapturing a multi-dimensional image of an eye; a display for displayinga representation of the multi-dimensional image of the eye, a networkinterface; a robotic arm configured to position the surgical camera; anda surgical suite optimization engine (SSOE) communicatively coupled withthe surgical console and the robotic arm, wherein the SSOE include anSSOE processor and an SSOE memory containing SSOE instructions which,when executed by the SSOE processor, cause the SSOE processor totransmit instructions, after receiving the action code from the surgicalconsole, which: cause the robotic arm to position the surgical camera onan area of the eye that is a subject of the subsequent procedure stage;and cause the surgical camera to focus on an area of the eye that is thesubject of the subsequent procedure step.
 43. The ophthalmic system ofclaim 42, wherein the SSO memory contains additional instructions, whenexecuted by the SSOE processor, cause the SSOE processor to transmitinstructions, after receiving the action code from the surgical console,which retrieves user preference data describing a user-definedadjustment in one or more of the plurality of procedure stages.
 44. Theophthalmic system of claim 42, wherein the display automatically adjustsdisplay parameters of the stereoscopic representation of themulti-dimensional image of the eye after receiving the action code fromthe surgical console.
 45. The ophthalmic system of claim 44, whereinautomatically adjusting display parameters further involves displayingone or more overlay over the representation of the three-dimensionalimage of the eye.
 46. The ophthalmic system of claim 44, whereinautomatically adjusting display parameters further involves filteringcolor attributes of the representation of the multi-dimensional image tohighlight one or more areas of the eye in the representation of thethree-dimensional image.
 47. The ophthalmic system of claim 46, whereinautomatically adjusting display parameters further involves the displayprocessor enhancing image features of the stereoscopic representation ofthe three-dimensional image.
 48. The ophthalmic system of claim 1,further comprising a laser treatment component configured to emit anaiming beam and a treatment beam, wherein the laser treatment componenttransitions between an emission of the aiming beam and an emission ofthe treatment beam after receiving the action code from the surgicalconsole.
 49. The ophthalmic system of claim 48, wherein the SSOE isfurther configured for performing digital signal processing toneutralize a green flashback from a retina after emission of thetreatment beam.